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Launched in 1995, the American Council of Trustees 
and Alumni (ACTA) is an independent, non-profit 
organization dedicated to working with alumni, donors, 
trustees, and education leaders across the country to 
support liberal arts education, high academic standards, 
the free exchange of ideas on campus, and high-quality 
education at an affordable price. 

ACTA’s Institute for Effective Governance, founded in 
2003 by college and university trustees for trustees, is 
devoted to enhancing boards’ effectiveness and helping 
trustees fulfill their fiduciary responsibilities fully and 
effectively. IEG offers a range of services tailored to 
the specific needs of individual boards, and focuses on 
academic quality, academic freedom, and accountability. 


Is it possible to measure intellectual growth in 
college? To know whether students graduate 
with core collegiate skills in analytical reasoning, 
effective writing, and mathematics? Can you 
measure the value that your institution’s general 
academic program adds during a student’s college 
years? 

Yes, you certainly can. There are readily available 
tests in wide use around the nation that are 
inexpensive and which allow easy benchmarking 
for results. They are valid, that is to say, they yield 
results that are true for the subjects they measure 
and reliable, that is to say, they yield consistent 
results. They reveal whether an institution as a 
whole is effective in increasing student learning and 
can even be used to show an individual student’s 
progress through core collegiate skills. Since they 
are nationally-normed with the average results of all 
institutions in America that used the test in a given 
year, your institution can, if it desires, interpret its 
scores in light of institutions across the nation. 

In other words, there are powerful tools that your 
institution can use in a number of ways to measure 
effectiveness in the single most important thing you 
do: educating students. 






If higher education has the courage to take responsibility 
for honestly assessing student learning and publishing 
the results, the measuring stick will be a tool. If it doesn't, 
the stick could easily become a weapon. The time for 
making that choice is drawing to a close. 

Kevin Carey, Director of Education Policy, 
New America Foundation 
“Student Learning: Measure or Perish” 
Chronicle of Higher Education, Dec. 12, 2010 

llllllllllllllllllllllllllllll lllllllllllll II I II : 

Why Measure General Collegiate Skills? 

Almost every professor gives quizzes and tests and 
assigns a final grade to each student. Why does 
your institution need to look further for evidence of 
student learning gains? 

First, most professors are deeply devoted to 
their academic specialty. They normally take 
responsibility for their classes and their subjects, 
and only rarely for the overall intellectual progress 
of the students. One might hope that the English 
department would be concerned if students 
in its classes had only a grade-school grasp of 
mathematics or were scientifically illiterate. One 
might hope that engineering professors would 
firmly correct students for poor grammar and 
writing style. But those expectations, however 
reasonable, are commonly found to be unrealistic. 

Examined more broadly, evidence is mounting 
that a number of colleges must do a better job of 
ensuring that graduates have the basic collegiate 
skills that employers expect. Demands from the 
public, the press, and policymakers for better 
results are increasing — rapidly, too. 


The National Adult Literacy Survey and the 
National Assessment of Adult Literacy, conducted 
by the U.S. Department of Education in 1992 
and 2003, revealed that most college graduates 
fall below proficiency in verbal and quantitative 
literacy. They cannot reliably answer questions that 
require comparison of viewpoints in two different 
editorials or compare the cost per ounce of food 
items. These shocking findings were confirmed in 
2006 with an analysis conducted by the prestigious 
American Institute for Research. 1 

Then in 2011, the University of Chicago Press 
published Academically Adrift: Limited Learning on 
College Campuses, by Professor Richard Arum of 
New York University and Professor Josipa Roksa of 
the University of Virginia. Working with the Social 
Science Research Council, these distinguished 
sociologists tested over 2,300 college students at 
24 accredited institutions, and their findings have 
rattled the nation. 45% of the students showed 
no significant intellectual gains after the first two 
years of college, and 36% showed no improvement 
after four years. The study showed that “high- and 
low-performing students can be found at each 
institution and within each level of selectivity.” 2 

Make no mistake about these findings: a student 
can pay the earth in tuition and leave with a 
diploma, but without a college education. 

Responsible colleges should not play dice with 
students’ futures — or with the hard-earned money 
of students, their families, and taxpayers. That’s 
why it’s important to measure student learning 






and adopt appropriate strategies, as necessary, to 
improve teaching and learning on their campuses. 

Responsible colleges and universities create 
a culture of assessment on which they craft 
academic policy to improve student learning. 

Years ago, an undistinguished regional university, 
Northwest Missouri State, transformed itself into 
the academic powerhouse now known as Truman 
State University, and its drive to excellence was 
powered by careful assessment. As one Truman 
State professor wrote, “The key is whether the 
numbers shake you out of your complacency. We 
all thought we were good before, but the numbers 
convinced us that we needed to make changes 
in the curriculum and in the way we designed 
student questions and assignments. ” Assessment 
information enabled the faculty to decide upon 
curricular change more swiftly than they might 
have otherwise. 3 

The key is whether the numbers shake you out of your 
complacency. We all thought we were good before, 
but the numbers convinced us that we needed to make 
changes in the curriculum and in the way we designed 
student questions and assignments. 

Truman State professor cited in 
USA Today, Feb. 2, 1987 

llllllllllllllllllllllllllllllllllllllllllllllllll 

Three Major Assessment Instruments 

At this time there are three assessments in wide use 
to measure academic progress in core collegiate 
skills. Many institutions choose a single one, 
though some colleges and universities test cohorts 


of students with different instruments. All three 
of these tests can be used to show the value- 
added factor of a college education and to show 
attainment of skills relative to other institutions. 
Many institutions for accountability purposes 
choose to report the results on their websites; 
others use the information for institutional 
planning and diagnostic purposes. Increasingly, 
institutions use general education testing to cast 
light on the effectiveness of particular educational 
practices; some institutions also use these tests to 
identify academic weaknesses in individual students 
in order to develop appropriate interventions. 

The Collegiate Learning Assessment (CLA ) 4 

began development in 2000 under the auspices of 
the Council on Aid to Education, which was then a 
subsidiary of the Rand Corporation; the instrument 
was piloted in 2002-03. Its supporters include the 
Carnegie Corporation, the Teagle Foundation, 

The Bill and Melinda Gates Foundation, the Ford 
Foundation, and the Lumina Foundation. At this 
time, over 500 public and private institutions and 
250,000 students have participated in the CLA; 
over 80% of the public universities participating 
in the Association of Public and Land Grant 
Universities’ Voluntary System of Accountability 
have chosen the CLA as their assessment system. 
The CLA is the instrument that was used for 
the investigations that led to the publication of 
Academically Adrift and continues to inform the 
educational research of the Social Science Research 
Council. In 2012-13, it will become the assessment 
instrument of choice for the outcomes assessment 






project of the 3 1 nations in the Organization of 
Economic Cooperation and Development. 

Institutions may choose between a low-cost 
assessment of institutional effectiveness, in which a 
cross-section of 100 freshmen and 100 seniors take 
the test, or they may choose to sample in-depth 
within their student population to determine how 
relative subgroups (e.g., demographically, choice 
of major) contribute to the overall institutional 
performance. The standard administration of the 
CLA costs $6,600 and can identify a difference in 
performance between the freshmen and seniors 
down to half a standard deviation, yielding highly 
useful insights. The CLA has also developed a 
series of Performance Task Academies whose 
purpose is to train faculty and administrators 
to construct college courses that have clear and 
measurable positive effects on core collegiate 
skills. More than 1,000 individual faculty and 
administrators have attended Performance Task 
Academies run by the CLA. 

The CLA differs from other general education 
assessments in that it is based entirely on 
performance tasks that replicate the challenges of 
“real world” employment. An example provided 
by the CLA of a typical assessment exercise can be 
seen in Ligure 1. 

Although the CLA is widely used as a measure of 
institutional accountability for academic progress, 
it has a wider purpose in teaching and learning. 

In the words of Roger Benjamin, the president of 
the Council on Aid to Education, “The principal 


Figure 1. 

Typical CLA Assessment Exercise 

You are the assistant to Pat Williams, the 
president of DynaTech, a company that 
makes precision electronic instruments 
and navigational equipment. Sally Evans, 
a member of DynaTech’s sales force, 
recommended that DynaTech buy a small 
private plane (a SwiftAir 235) that she 
and other members of the sales force 
could use to visit customers. Pat was 
about to approve the purchase when there 
was an accident involving a SwiftAir 235. 

You are provided with the following 
documentation: 

1. Newspaper articles about the 
accident 

2. Federal Accident Report on in-flight 
breakups in single engine planes 

3. Pat’s e-mail to you and Sally’s e-mail 
to Pat 

4. Charts on SwiftAir’s performance 
characteristics 

5. Amateur Pilot article comparing 
SwiftAir 235 to similar planes 

6. Pictures and description of SwiftAir 
Models 180 and 235 

Please prepare a memo that addresses 
several questions, including what data 
support or refute the claim that the type 
of wing on the SwiftAir 235 leads to more 
in-flight breakups, what other factors 
might have contributed to the accident 
and should be taken into account, and 
your overall recommendation about 
whether or not DynaTech should purchase 
the plane. 






goal of the CLA is to assist faculty in improving 
teaching and learning.” In 2012, the CLA released 
an individual student-focused version of the test, 
designed to provide fine-grained institutional 
accountability measures, as well as a diagnostic 
instrument and credential for individual student 
achievement. Applications for the new instrument 
include documentation of high school proficiencies 
for college admissions and certification of 
proficiency in higher order thinking skills that 
could be used along with the student’s transcript as 
a credential for prospective employers. 

The Proficiency Profile 5 is administered by 
Education Testing Service (ETS). The ETS 
Proficiency Profile is the successor to the Academic 
Profile and the MAPP (Measurement of Academic 
Progress and Proficiency). Administration of the 
test costs $16.50 per student for both the paper 
and online format, with discounts for numbers 
greater than 500. Over 400 institutions have used 
the Proficiency Profile or an earlier version of the 
instrument. It measures skills in reading/critical 
thinking, writing, and mathematics in a multiple- 
choice format, and offers an optional essay module. 
It is available in two versions: a two-hour exam 
and a forty-minute version. “The ETS Proficiency 
Profile has the student demonstrate knowledge, 
comprehension, application and synthesis of 
a variety of skills. Such an approach allows 
institutions to address the complex demonstrations 
of skills. ... Today, these levels of proficiency ... are 
equated with accountability in student learning,” 
says Dr. Hazel A. Ervin, Director of General 


Education and associate professor of English at 
Morehouse College. Other institutions that use the 
Proficiency Profile include Clemson University 6 ; 
Oklahoma State University 7 ; and Florida 
Agricultural and Mechanical University 8 , which 
seeks to test all incoming freshmen and graduating 
seniors with this instrument. 

Figure 2. 

ETS Proficiency Profile Sample Question 

From 6:00 a.m. to 7:00 a.m. one 
morning, the temperature rose 7° F From 
7:00 a.m. to 8:00 a.m., the temperature 
rose 2° F, and from 8:00 a.m. to 9:00 
a.m., the temperature rose 3° F If the 
temperature was 55° F at 9:00 a.m., what 
was the temperature at 6:00 a.m.? 

(A) 43° F 

(B) 53° F 

(C) 57° F 

(D) 67° F 


Collegiate Assessment of Academic Proficiency 
(CAAP ) 9 is administered by ACT. The CAAP has 
six modules: reading, writing, mathematics, science, 
and critical thinking, plus an essay module: the 
first five modules are multiple-choice tests. Each 
module takes forty minutes to complete. The five- 
module multiple-choice test costs $21 per student, 
with an additional $14 fee for the essay module, 
with discounted fees for administrations of over 
500 tests or for a single multiple-choice module. 
Institutions may choose which of the modules they 






wish to use. Nearly 500 colleges and universities 
have used the CAAP. 

The CAAP forms an important part of Truman 
State’s storied assessment system: for many years, 
half of the freshman class took the CAAP, which 
was then administered again to half of the junior 
class, while the other half used the forerunner of 
the ETS Proficiency Profile. 

Since 1998, all South Dakota public colleges 
and universities administer the CAAP to all 
students, and it has been a major diagnostic tool 
for identifying areas of academic weakness, both 
individual and institutional. The South Dakota 
Board of Regents reports: 

“After one university fell below the national mean 
in English five out of six years, that university 
implemented an additional writing requirement for 
all of its students. Another university has placed 
a new focus on math after receiving lower than 
expected scores in that subject.” 

Before CAAP, we really did not have the mechanism 
other than anecdotes to show that our universities have 
increased the knowledge of students. But if you want to 
use data-driven decision making, anecdotes just don't 
do it. 

Lesta Turchen, Chief Academic Officer, 
South Dakota Board of Regents 

lllllllllllllllllllllllllllllllllllllllllllllllll 

According to Lesta Turchen, the Board of Regents 
chief academic officer at the time the assessment 
initiative began, “The CAAP exam has become a 


primary tool for working with individual students 
to improve their skill levels so that they can be 
successful in upper- division courses in their majors 
If you can better prepare students to be successful 
in their coursework, the students themselves are 
more likely to reach their personal goals.” 


Figure 3. 

CAAP Sample Question 

Writing Skills Sample Passage and Items 

In the end, everyone gives up jogging. 
Some find that their strenuous efforts to 
earn a living drains (1) away their energy. 

1. A. NO CHANGE 

B. drain 

C. has drained 

D. is draining 

Others suffering from (2) defeat by 
the hazards of the course, from hard 
pavement to muddy tracks, and from 
smog to sleet and snow. 

2. A. NO CHANGE 

B. suffered 

C. suffer 

D. suffering with 

These can also (3) collapse in their 
sneakers. 

3. A. NO CHANGE 

B. Still others 

C. They can also 

D. They also can 






Challenges and Solutions for Implementation 

It is rare that accountability measures do not 
encounter resistance, and assessment of general 
education is no exception. Students usually face 
no consequence for poor performance on these 
assessments and some will not take the examination 
as a serious exercise. The testing companies offer 
a variety of suggestions for improving the accuracy 
of the results, including prizes for high scores 
and other incentives for good-faith participation. 

At institutions where individual performance 
is monitored and serves as a potential gateway 
to upper-level courses, there is significantly 
less difficulty in ensuring serious participation. 
Building a faculty commitment to assessment is 
also a challenge: some institutions have established 
prizes for departments whose majors can show 
particularly strong gains in collegiate skills via 
general education assessments or other objective 
assessments of intellectual growth. The Association 
of Public and Land Grant Universities (APLU) and 
the Council of Independent Colleges (CIC) have 
helped to encourage the faculty and administration 
of their many member institutions to use the CLA, 
the CAAP, or the Proficiency Profile. 

You may hear the argument that using a 
standardized measure of general education 
progress will encourage “teaching to the test.” In 
the case of the CLA — which has no multiple choice 
questions — such an argument is unsustainable: 
what, after all, could be wrong about helping 
students deepen the cognitive skills that enable 


them to do well on an assessment that imitates 
the intellectual challenges of the workplace? If 
the instrument tests meaningful and appropriate 
skills and knowledge — and that is true of the CLA, 
CAAP, and Proficiency Profile — a reasonable 
counter-question is why any professor or 
administrator would be afraid to have such basic 
information about student progress. 

You may hear the argument that a single test such 
as the tests mentioned above cannot provide a 
clear picture of the academic effectiveness of the 
institution. That is true: you will also want to know 
the results of professional licensure examinations, 
average GRE scores, and other measures of 
academic progress. No single medical test tells a 
doctor everything about a patient’s health. But 
that is not an argument against using medical 
diagnostics that are routine procedure in quality 
medical care. 

You may be urged to adopt a portfolio assessment, 
but it is important not to confuse quantifiable, 
nationally-normed results with a portfolio 
assessment. A portfolio assessment is based on 
selecting the “best” performances to include, 
rather than documents that are representative 
of the abilities of the candidate. They lack the 
reliability and validity that the standardized general 
education assessments have. And in the crucial 
area of formative assessment, it is difficult to make 
decisions about how to improve teaching and 
learning relying on portfolios. There is, in short, a 
“fudge factor” that a portfolio allows: it is a system 






Referenced Websites 


that is not well suited for accountability or even 
objective measurement. 

In sum, higher education now has several cost- 
effective measurement tools that show how well 
students are gaining the core skills that should 
be expected in a high-quality degree program. 
Governing boards, faculty, and administrators all 
have a stake in objectively measuring the academic 
strength of their institution and then building on 
that strength and remedying any programmatic 
weaknesses. Objective data that allow comparison 
from year to year will benchmark progress, and 
help the school continuously improve teaching and 
learning. 

Let ACTA know if we can advise and help 
you understand and implement this crucial 
measurement of academic progress. 
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